Learning Social Relations from Videos: Features, Models, and Analytics

Despite the progress made during the last decade in video understanding, extracting high-level semantics in the form of relations among the actors in a video is still an under-explored area. This chapter discusses a streamlined methodology to learn interactions between actors, construct social networks, identify communities, and find the leader of each community in a video sequence from a sociological perspective. Specifically, we review one of the first studies reported in [8] toward learning such relations from videos using visual and auditory cues. The main contribution can be stated as the association of low-level video features to social relations by means of machine learning mechanisms, including support vector regression and Gaussian processes. The resulting social network is then analyzed to find communities of actors and identifying the leader of each community, which are two of themost important tasks in social network analytics. Furthermore, as an extension to the basic framework, we discuss the relationship between visual concepts and social relations that has been explored in [9]. In this setting, visual concepts serve as mid-level visual representation in inferring social relations and are compared with features employed in the basic framework. Recently, researchers have devoted countless efforts on understanding the scene content from video by analyzing the object trajectories and finding common motion patterns [2, 7, 13, 22, 23, 44]. Most of these efforts, however, did not go beyond analyzing or grouping trajectories, or understanding individual actions performed by tracked objects [3, 16, 20, 36, 43]. The computer vision community, generally